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Abstract 


In low-income countries, private schools are perceived as 
superior alternatives to the public sector, often improving 
achievement at a fraction of the cost. It is unclear whether 
private schools are as effective in middle-income countries 
where the public sector has relatively more resources. To 
address this gap, this paper takes advantage of lottery-based 
admissions in first grade for a Mexico City private school 
that targets and subsidizes attendance for low-income chil- 
dren. Over three years, selected students via lottery scored 
0.21 standard deviation higher than those not selected in 
literacy tests, corresponding to a normalized gain of one- 
half of a grade level every two years. Lottery winners also 


statistically outperformed those not selected in math, but 
the gains were more modest. Relative to the control group, 
parents of selected students were more satisfied with their 
school and had higher educational expectations for their 
children. Unlike findings from low-income countries, 
these gains came at increased cost—twice as much on a 
per pupil basis relative to public schools. Additional anal- 
yses indicate gains made by the lowest income students in 
the sample help explain the school’s impact. This suggests 
private schools could bring down persistent achievement 
gaps in these countries, but puts into question the validity 
of implementation at scale. 
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1. Introduction 

In low-income countries, private schools are perceived as superior alternatives to 
the public sector, often improving achievement at a fraction of the cost (e.g., Patrinos, 
Barrera-Osorio and Guaqueta 2009; Muralidharan and Sundararaman 2015; Andrabi, Das 
and Khwaja 2008). Many of these countries have weak public school systems, since 
public sector capacity is strongly correlated with national income (Rauch and Evans 
2000; DeJanvry and Dethier 2012). It is unclear whether private schools are as effective 
in middle-income countries where the public sector has relatively more resources. 

This paper analyzes the academic impact and cost of an innovative private 
schooling model in Mexico. We take advantage of lottery-based admissions in first grade 
for a Mexico City private school that targets and subsidizes attendance for low-income 
children. Over three years, students selected via lottery score 0.21 SD higher than 
students not selected in literacy tests, corresponding to a normalized gain of about one- 
half of a grade level every two years. Lottery winners also statistically outperform those 
not selected in math, but gains are more modest. These gains come at an increased per- 
pupil cost--more than double than traditional public schools. The school we study targets 
children in poverty. Our findings confirm that this student population benefits the most. 
To arrive at this result we re-weight the sample to resemble an average Mexico City 
household in terms of income and other socio-economic characteristics. Other results 
suggest parents of children attending the private school are more satisfied and more likely 
to expect that their children will go to college than parents in traditional public schools. 

There are only a handful of rigorous studies exploring the effects of private 


schooling in middle-income countries. The most well-known school choice program in 


the region, the Chilean national voucher reform, significantly increased student sorting 
and stratification by socioeconomic status with mixed results on test scores (see, for a 
review, Epple, Romano and Urquiola 2017), but positive effects on subsequent labor 
market outcomes (Bravo, Mukhopadhyay and Todd 2010; Patrinos and Sakellariou 
2011). In Colombia, a voucher program which allowed low-income students to attend 
private secondary schools showed positive effects on high school graduation, college 
access and the labor market (Angrist, Bettinger and Kremer 2006; Bettinger, Kremer and 
Saavedra 2010; Bettinger et al. 2018). Most of these studies, however, focus on private 
school vouchers rather than private schools, and on student outcomes in secondary 
school. 

The majority of students in middle-income countries, however, do not 
successfully transition from primary into secondary school. In Mexico, for example, only 
half the students who complete elementary school ever attend high school (Kattan and 
Szekely 2015). To understand the potential benefits of private schools in these countries, 
it is important to look at early grades. Establishing a good literacy and numeracy 
foundation in the early grades will make the transition to secondary school less risky and 
eventually lead to long-term academic success (Duncan et al. 2007; Hernandez 2011; 
Cunningham and Stanovich 1997; Watts et al. 2014). To the best of our knowledge, there 
are no published experimental studies that investigate the academic effects of private 
schools and costs in middle-income countries in the earlier grades. 

Our results meet high evidence standards. Overall and differential sample attrition 
in the study are low and well within acceptable benchmarks for experimental studies, 


such as the Institute of Education Sciences' What Works Clearinghouse (Deke, Sama- 


Miller and Hershey 2015). Our findings are robust to various empirical strategies, 
including randomization inference and re-weighting for increased external validity. In 
addition to leveraging the experimental, lottery-based research design to demonstrate 
academic impacts, we conduct additional observational analyses to understand mediating 
mechanisms. Using a strategy that matches on baseline covariates, we find that the length 
of the school day partially mediates the school's effects on achievement. 

Our findings are based on the results of a single school and families applying to 
Christel House de Mexico (CHM) do not resemble the average family in Mexico City. To 
account for this, we re-estimate all models using a method of entropy weights, which 
calibrates each observation based on their income and other socio-economic 
characteristics. Using this information, we generate two re-weighted samples: (1) one that 
resembles the average Mexico City household and (2) one that resembles a low-income 
Mexico City household (i.e., those with incomes below the median). This analysis 
indicates that CHM effects are no longer significant when the population resembles an 
average family in Mexico City, but are significant when the study sample resembles a 
low-income family. In other words, re-weighting the units so that study students look 
more like the average student in Mexico City mitigates our findings and highlights the 
fact that CHM is a model that is particularly beneficial for the most disadvantaged 
children. 

Our findings are consistent with similar studies conducted in the United States, 
which suggest private schools are especially favorable for low-income students with poor 
public schooling alternatives (Chabrier, Cohodes and Oreopoulos 2016; Angrist et al. 


2012 for KIPP Schools; Dobbie and Fryer 2011 for Harlem Children's Zone). Our 


contribution to the debate about the value of private schools in developing countries is to 
suggest that, at least in the context of middle-income countries like Mexico, expanding 
private schools could improve outcomes for the most vulnerable children, but will not 
necessarily save taxpayer money. The results presented in this paper indicate that 
significantly improving learning outcomes for low-income children in these countries 
requires substantial additional resources. 

2. Background 

Close to 90 percent of elementary and lower secondary students in Mexico attend 
public schools (OECD 2014). Learning outcomes in public schools are quite poor: close 
to 60 percent of 15-year old students in the country place below basic proficiency levels 
in the Math PISA tests (OECD 2015). Results on national standardized tests paint a 
similar picture: two-thirds of 9th graders test below basic standards in mathematics and 
one-third test below basic standards in Spanish (INEE 2018).” Results for students in 
rural schools (community schools and "telesecundarias") are particularly dismal: more 
than 70 percent of students in rural middle schools test below the basic proficiency level 
in math, and close to 50 percent tests below this level in Spanish. 

Most public schools in Mexico meet for half a day (4.5 hours) and follow the 
Ministry of Education curricula. Most offer some kind of preventive health services, 
remedial and extracurricular activities. None of these services are required by law. There 
are no contractual parental commitments, although public school parents in Mexico 
routinely perform volunteer cleaning and maintenance services (Santibafiez, O'Donoghue 


and Abreu 2014). Teachers are hired centrally by the federal (or state) government and 


2 Results refer to the "PLANEA" test. This test was introduced in 2015 to replace "ENLACE" which was 
administered from 2006-2013. 


are granted tenure after 2 years on the job. Over 90 percent of national expenditures on 
education go toward paying staff payroll (OECD 2013). To become a public school 
teacher, teachers must have a teaching credential, which is granted upon completion of a 
B.A. teacher education program. Public schools in Mexico can be elementary (grades 1- 
6), middle (grades 7-9) or high schools (grades 10-12). 

In 2007, owing to declining demographic trends in many large cities, the Ministry 
of Education began converting half-day elementary schools into full day schools. In 
2016, there were over 24,000 full-time schools operating across Mexico. About 10 
percent of these new full-time schools are located in Mexico City and its metropolitan 
area. Full time schools operate as 6-hour schools without lunch ("Escuelas de Jornada 
Ampliada") or 8-hour schools with lunch ("Escuelas de Tiempo Completo"). Research on 
full-day schools in Latin America finds positive effects on students, particularly those in 
rural and low-income communities (Cabrera-Hernandez 2016; Padilla-Romo 2016; 
Holland, Alfaro and Evans 2015). 

To enroll in a public school in Mexico City, parents submit an application to the 
Administracion Federal de Servicios Educativos del Distrito Federal (AFSEDF), Mexico 
City's office of education. Parents must choose schools in their residential or work area. 
Charter schools are not legally allowed, and neither are vouchers at any educational level. 
Privatization efforts are highly controversial in Mexico due to formidable opposition 
from powerful teachers' unions (Santibafiez and Jarillo 2008; Hecock 2014). This is not 
unique to Mexico. In low- and middle-income countries public delivery dominates 
education provision and accounts for over 85 percent of total primary school, and 75 


percent of secondary school enrollment (World Bank 2018). 


Christel House de Mexico (CHM) 

CHM is part of the Christel House International Organization, a philanthropic 
educational initiative that provides educational services to socioeconomically 
disadvantaged children. Christel House International operates two schools in India, one in 
South Africa, four charter schools in the U.S., and one school in Mexico. For this study 
we collaborated with the school located in Mexico City. During the time of our analysis, 
the school served grades | to 9, but has since expanded to serve grades | to 12. The 
school provides free or heavily subsidized education to over 400 low-income children. 
Most children belong to families with disadvantaged circumstances that might include 
infrequent employment, disability, incarcerated parents, mental illness, homelessness, 
and single parenthood. In 2013, the last year that Mexico held a national standardized 
exam testing all students in all grades, CHM 3rd graders scored 661 points in 
Mathematics, and 598 points in Spanish. This compared favorably to the average for 
Mexico City public schools of 619 points in Mathematics (std. dev. = 56) and 586 points 
in Spanish (std. dev. = 71).° 

A small proportion of CHM families pay tuition on a sliding scale according to 
household income. In our sample, 10 percent of the families who won the lottery and 
enrolled in CHM paid tuition. Parents who reported paying tuition paid US$10-$20 per 
month on average.* The remaining 90 percent of the families do not pay any tuition. 
Information from a parent cost survey administered as part of this study revealed that 


Christel House International covers CHM's administrative and fundraising expenses. The 


3 Own calculations based on ENLACE Mexico City data for 2013. Available at: 
http://www.enlace.sep.gob.mx/ba/resultados_historicos_por_entidad_federativa/. 

4 At 2016 exchange rate of 20 pesos per US$. As a comparison, average elementary private school tuition 
in Mexico City is about $150 per month. See: http://izq.mx/noticias/3 1/2018/01/cual-es-el-costo-de-tener-a- 
tus-hijos-en-una-escuela-privada-son-gastos-millonarios/. 


school's operation, including teacher salaries, scholarships, nutrition, extracurricular 
activities, school uniforms, etc., is funded through local business and individual tax- 
deductible contributions and charitable donations. The school receives a small amount of 
public subsidies to pay for school lunches. 

Like many charter schools in the U.S., CHM requires strong parental involvement 
and commitment. Parents sign an agreement at the beginning of the year agreeing to 
attend 12 sessions of parenting education and donate 40 hours a year to school duties. For 
the most part, these duties include cleaning and maintenance. Students commit to an 
attendance rate of at least 95 percent a year. CHM has high levels of compliance with 
parent-student-school agreements and attendance is very high. Few students have ever 
been asked to leave for non-compliance reasons. 

CHM's educational model targets cognitive (academic) and non-cognitive 
development. Relative to most public schools in Mexico City, CHM offers a longer 
school day—9 versus 4.5 hours in most public schools (see Appendix Table A1). Close to 
six hours per day are spent in academic activities and close to two hours per day are spent 
on enrichment and other activities (sports, music, art, character development). 

CHM follows the curriculum and content standards set by the Ministry of 
Education in Mexico. They use publicly provided textbooks—which are provided free of 
charge to all schools (public and private) in Mexico. CHM purchases additional 
textbooks and resources from commercial curriculum vendors (i.e., Pearson). The school 
offers a wide range of academic and enrichment activities including sports, arts, English 
as a second language, a tutoring program for students needing remediation, counseling 


services for career- and college-readiness, and computer skill development. Consistent 


with CHM’s "holistic" pedagogical approach, students receive preventative health 
services—mental health services, annual medical and dental check-ups—and a 
comprehensive nutritional package that includes daily breakfast and lunch following 
healthy dietary guidelines. All children receive school uniforms and school materials free 
of charge. 

To ensure high-quality teaching, CHM carefully selects teachers and dismisses 
them if they do not perform up to their standards. Teachers are not unionized and do not 
have tenure --as they do in public schools. Teachers receive professional development 
and are monitored and supported by the elementary school coordinator. In addition, the 
school employs literacy and mathematics coordinators to coach and support teachers in 
these areas. All students are expected to finish high school and go to college. Table 1 
presents a contrast with curricular and program offerings at Mexico City public schools. 
CHM offers a much longer school day than traditional public schools, and a slightly 
longer day than full-time public schools. The school offers more nutrition, enrichment 
and other services. Overall, teacher seniority is lower than at public schools and teachers 
can be hired/fired at will. Teacher salaries are also lower at CHM than in public schools. 


Class sizes are the same. 


Table 1. Christel House de Mexico and Public Schools - Main Features 


CHM Traditional Extended Full-Time 
Public Day Public Public 
Schools Schools Schools 


" Jornada "Escuelas 
Ampliada" Tiempo 


Completo" 
Length of school day (hours) 9 4.5 6 8 
Curriculum National + National National National 
other ’! 
English as a second language, Yes Yes Yes Yes 
Breakfast Yes Yes” Yes” Yes” 
Lunch Yes No No Yes 
Art, music, computer skills Yes In some Yes Yes 
schools 
Autonomy over teaching Yes No No No 
hiring/firing 
Teacher salaries + benefits (average $16,829 $18,119 $18,119* $18,119* 
monthly, 1*-3" grade only) in 
Mexican Pesos” 
Teacher Seniority (average, 1-3" 2 8 8 8 
grade) (average) 
Parenting programs, psychological Yes No No No 
services for students/parents 
Extended day program, tutoring, Yes In some Yes Yes 
socio-emotional development, schools 
health preventive services. 
Class Size (average) 27 27 27 27 


/1 CHM uses the national curriculum provided by SEP, but supplements with other textbooks, instructional materials 
and resources. 

/2 Only in schools that participate in the National Free Breakfast Program. These are usually schools that enroll a 
majority of low-income students. 

/3 Salaries are for a 4.5 hour (traditional) teaching position. In schools where teachers teach an extended day, or 
multiple positions, salaries will be higher. 

Source: For CHM, cost interviews conducted as part of this study. For public schools, various published sources and 
interviews with a small sample of schools. See cost section for details. 


3. Data and Research Design 
Lottery Procedures and Data 
Every year CHM has more eligible applicants than open seats for its 1st grade 


cohort. Before this study began, CHM agreed to hold a public lottery for available first- 


10 


grade seats. We designed the lottery and carried it out. Figure 1 shows a timeline of the 
study, including baseline data collection, lottery, and testing. 


Figure 1. Timeline of Main Evaluation Events 


Jul/14 Nov-Jan/16 
Lottery Follow-up 
(Cohort 2) Parent 
Survey 
Jun/13 Jun/15 
Baseline May-Jun/14 Baseline Sep-Nov/16 
Data EGRA/ Data Qualitative 
Collection EGMA Collection Study 
“t" 1) i a oa 5 3) 
2013/14 2014/15 2015/16 2016/17 
| Jun/14 May-Jun/14 
Jul/13 Baseline Data EGRA/ May-Jun/16 
Lottery Collection EGMA EGRA/ 3 
(Cohort 1) (incl. Parent application EGMA pence 
eae Jul/15 application Collection 
(Conor) Lottery 
(Cohort 3) 


We collected data on lottery results (including wait list assignments), and 
enrollment. Each year, we updated the data for both CHM and public schools to account 
for any sample attrition. Between 2013 and 2015, 242 students participated in the lottery 
as part of three cohorts.° CHM carried out separate lotteries for boys and girls, effectively 
stratifying by gender, to preserve a balanced gender mix. Through the lottery mechanism, 
100 students, belonging to three cohorts, were offered a seat in CHM's Ist-grade cohort 
and 142 were not offered admission. Of the students offered a spot, 50 percent were girls 
and 50 percent were boys. In each lottery cohort, CHM also randomly selected 10 


students (5 boys, 5 girls) for a wait-list. Children in the random wait list were offered a 


5° Applicants with siblings already enrolled in CHM have guaranteed admission and did not participate in 
the lottery. They are not reflected in these numbers. Two parents/students refused informed consent to 
participate in this study. While they still participated in the lottery, they are not included in this sample. 
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spot whenever a lottery winner did not enroll in CHM or dropped out. For the analysis, 
we classify wait-listed children as treated. Overall, treatment compliance was high. Of 
those originally offered a Ist grade spot (and not on the waitlist), 94 percent took up the 
offer and enrolled in CHM. Of those originally on the waitlist and subsequently offered a 
spot, 9 enrolled in CHM. This represents 6.5 of the control group. However, as 
previously noted, we count waitlisted children—regardless of subsequent enrollment in 
CHM-—as treated. 

Data were collected at baseline (before the lottery) and at the end of each 
academic year. We followed each cohort from the time of the lottery until the last data 
collection round in 2015. This means that for the first cohort we have three years of data 
(i.e. data for Ist, 2nd and 3rd grades), for the second cohort we have two years of data 
(1st and 2nd grades), and for the third cohort we have one year of data (1st grade). In our 
tables and discussion, we refer to each of these points as Ist, 2nd, and 3rd follow-ups. 

In 2015 the Mexico City office of education (AFSEDF) did not allow test 
administrators access into control group public schools. During this year control group 
students were tested in their homes, while CHM students were tested at the school. 
Varying testing conditions could by themselves explain some of the differences in test 
outcomes between treatment and control. To account for this potential source of bias, we 
run all of our analyses for three samples: (1) “One-Year Sample," which includes first- 
year follow-up results only, (2) "Same Testing Conditions Sample," which includes only 
observations that were tested under the same conditions (excludes 2015); and (3) "All 


Observations," which includes the full sample of observations in all follow-up years. 
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Lottery Sample 

CHM families tend to earn about half what those in the average Mexico City 
household sample earn on a monthly basis—about 3,700 pesos or US $185 per month 
compared to 7,800 pesos or US $390 for the Mexico City sample (exchange rate 20 
pesos/dollar). Overall, CHM mothers have slightly less schooling than the household 
sample, but in both cases the average level of schooling completed is lower secondary 
school (9th grade). It should be noted that Mexico City has one of the highest levels of 
average educational attainment in the country. More students live in single-parent homes 
in the CHM sample, than in the Mexico City household sample, about 50 and 20 percent 
respectively. Lastly, CHM families are more likely than the average Mexico City family 
to rent their home, to not have a kitchen, to not be hooked to a public utility source for 
electricity, and to not have a landline (see Table 2). 

After the lottery was conducted, students enrolled in either CHM or a public 
school. Of the total sample, 43 percent enrolled in CHM, 26 percent enrolled in full-day 
public schools, and 30 percent enrolled in half-day public schools. An additional two 


percent enrolled in schools of unknown type (i.e., full-day or half-day). 
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Table 2. Descriptive Statistics and Balance 


Mexico 
CHM- CHM- CHM p-valof Standardi City HH 
Control Treatment  Diff.(C- C-T Diff. zed Diff. Sample 
Variable (1) (2) T) 3) (4) 3 (5) (6) 
Age (incoming Ist graders)’! 5.216 5.297 -0.081 0.223 -0.16 -- 
(0.047) (0.046) (0.066) 
Gender (incoming Ist 
graders) 0.577 0.508 0.069 0.289 0.14 -- 
(0.047) (0.044) (0.065) 
Raven Global Score 
(incoming Ist graders) 15.599 16.445 -0.846 0.101 -0.21 -- 
(0.377) (0.350) (0.515) 
WISC Global Score 
(incoming Ist graders) 25.087 23.844 1.244 0.361 0.11 -- 
(1.082) (0.851) (1.360) 
Monthly mean income (in 
MX pesos)? 3,664 3,389 274 0.412 0.12 7,812 
(226.23) (242.95) (335.35) (419.12) 
Mother's education 
(Secondary=4) 4.144 4.219 -0.075 0.426 -0.07 4.35 
(0.072) (0.061) (0.094) (0.073) 
Mother has elementary 
school or less 0.177 0.142 0.036 0.449 0.10 0.22 
(0.036) (0.031) (0.047) (0.030) 
Mother has high school or 
more 0.360 0.398 -0.038 0.547 -0.08 0.45 
(0.046) (0.043) (0.063) (0.036) 
Both parents are present in 
the home 0.495 0.528 -0.033 0.613 -0.07 0.75 
(0.047) (0.044) (0.064) (0.031) 
Home is rented 0.374 0.456 -0.082 0.197 -0.17 0.24 
(0.045) (0.044) (0.063) (0.030) 
Home has a kitchen 0.681 0.598 0.083 0.179 0.18 0.95 
(0.044) (0.043) (0.062) (0.016) 
Home is hooked to public 
electricity 0.811 0.787 0.024 0.644 0.06 1.00 
(0.036) (0.036) (0.052) (0.000) 
Home has landline 0.373 0.339 0.034 0.576 0.07 0.46 
(0.045) (0.042) (0.062) (0.035) 
Observations lll 128 239 270,000 


* H<0.01, ** p<0.05 ** p<0.10 


Notes: Source for Mexico City sample is ENIGH-2014 (tradicional). HH with children in grades 1-3 in public schools. 
Sample represents over 270,000 HHs in Mexico City. Source for CHM treatment and control is lottery sample. F-value 
tests the hypothesis that the coefficients on key demographic variables measured at baseline (age, Raven, WISC) on 
winning the lottery (treatment status) are all jointly zero. F-value from joint test: 0.71. p-value from F-test 0.54. 


/1 Student Age for CHM is collected between March-June in the school year previous to entering Ist grade. 


/2 Average monthly household income from wages and other fixed sources. 
/3 Standardized with respect to the control group standard deviation. 
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Balance in Baseline Covariates 

Baseline differences between lottery winners (treatment) and losers (control) in 
baseline demographic and cognitive measures are not significant for any of the variables 
used in the analysis (see Table 2, Columns 3-4). The F-value for the joint test of balance 
for all these observable characteristics has a p-value of 0.54. All the differences were less 
than the WWC standard for baseline equivalence of 0.25 SD of the control group mean 
(see Table 2, Column 5). Together, these results strongly suggest the study sample was 
balanced prior to the lottery rendering the experimental design highly internally valid. 
Differential Attrition 

Overall sample attrition in the study was low. The sample contains 502 
observations over three data collection waves. We are missing data for 18 of these 
observations (3.5 percent attrition rate). These 502 observations represent 242 students. 
We are missing some follow-up data for 16 of these students, or 6.6 percent. Most of 
these students missed only one test application but have data for the other applications. 
This means attrition by wave is lower than the overall reported attrition rate. Student 
attrition was 1.2 percent of the sample at the first follow-up, 4.9 percent of the sample at 
the second follow-up, and 6.4 percent of the sample at the third follow-up. 

Differential attrition between treatment and control groups can generate bias for 
the treatment effect estimates if those who leave the treatment group differ from those 
who leave the control group along dimensions related to the outcome of interest. In such 
a case, estimated program impact can be biased even if the lottery successfully generates 


comparable treatment and control groups at baseline. 
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We follow current practice in the analysis of field experiments (Duflo, 
Glennerster and Kremer 2006) and use two tests to determine whether attrition is likely to 
generate bias in program impact estimates. The first examines whether the fraction of 
applicants with missing outcome data differs between lottery winners and losers. For 
each applicant cohort, we propose to regress A;,, an indicator random variable for 
whether outcome data are missing for a given applicant, on lottery status and other 
previously defined covariates, as follows: 

(1) Aic = 8¢ + Ne + T Lic + Xic B+ Gic 

The null hypothesis of no difference in the probability of dropping out of the 
evaluation sample between lottery winners and losers corresponds to Hj: T = 0. Note that 
this analysis needs to be conducted for each outcome at each follow-up point since 
attrition patterns can vary across outcomes and over time. 

The second test examines whether the factors that predict attrition are the same 
between lottery winners and lottery losers. To implement this test, we modify equation 
(1) to include interactions between Z;, and the vector of covariates X;-: 

(2) Aic = Oe + Ip + T Lig + Xie B+ LigX ig + Ci 

and test the null joint hypothesis that all the parameters in the vector @ are zero: Hp: a, = 
Qz = ‘t+ = A» = 0, where m is the number of baseline characteristics in the vector X. 
Rejection of the null hypothesis on these two tests raises concerns about attrition bias. 

Table 3 shows results of these two tests of differential attrition. The first test 
(Panel A) finds significant differential attrition of -0.04 for the pooled sample. An 
analysis of differential attrition by year of the study finds statistically significant 


differential attrition at the 2nd follow-up (coefficient of -0.06), and no statistically 
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significant differential attrition at the Ist and 3rd follow-ups (coefficients of -0.01, and - 
0.06). As previously mentioned, during the 2nd year of the study we were denied access 
into control group schools by the Mexico City educational authorities, which negatively 
impacted our ability to reach as many control students as we did in the other two years of 


the study. 


Table 3. Differential Sample Attrition 


Pooled Ist 2nd 3rd 
Follow-up Follow-up Follow-up 


Test 1: Differential Attrition Test 
Coefficient on Lottery Admission 


variable (model with lottery strata fixed -0.04** -0.01 -0.06** -0.06 
effects only) (0.016) (0.017) (0.028) (0.046) 
Intercept (control group mean) 31.02* -12.52 92.66 0.13** 


(18.645) (9.514) (59.339) (0.059) 
Coefficient on Lottery Admission 


variable (model with lottery strata fixed -0.04** -0.02 -0.06** -0.06 
effects + student covariates) (0.016) (0.021) (0.026) (0.046) 
Intercept (control group mean) 25.03 -33.68 100.73 0.00 


(25.081) (27.353) (67.873) _~—« (0.219) 


Test 2: Predictors of Differential F-test p-value 
Attrition Test 
F-test (attrition predictors do not vary 0.256 0.923 0.582 0.643 


across treatment and control - model with 
lottery strata FE + student covariates + 
interactions) 


N 520 242 184 94 


*** y<0.01, ** p<0.05, * p<0.1 

Note: Lottery strata fixed effects included for gender and cohort. Student-level covariates include RAVEN and WISC 
test score, age, mother's education, mean monthly income, both parents present in the home, home is rented, home has 
a kitchen. 

Test 1: OLS regression with sample attrition as dependent variable, and lottery admission as the key independent 
variable (to test for differential attrition between treatment and control group). 

Test 2: Same as Test 1 but adding interactions between lottery status and covariates. We report the joint f-test of 
significance that attrition predictors do not vary across treatment and control. This model is estimated using OLS 
regression. 
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A second test of differential attrition examines whether factors that jointly predict 
attrition vary among lottery winners and losers. This test cannot be rejected at 
conventional levels (see Panel B of Table 3). Together, these results suggest that, if 
anything, the missing treatment and control group data in the pooled sample and second 
follow-up appear to be missing at random. These low differential attrition rates are well 
within the benchmarks issued by the Institute of Education Sciences' What Works 
Clearinghouse standards for evidence in randomized experiments.° 
Outcome Measures - Instruments 

Mexico has no national or state standardized tests in grades 1-3. For this reason, 
we use the Early Grade Reading Achievement test (EGRA; RTI International 2009) to 
assess student skills in Spanish literacy and the Early Grade Mathematics Achievement 
test to assess student skills in mathematics (EGMA; Mejia and Pflepsen 2012). For this 
study we used RTI-approved Spanish translations of EGRA, which we adapted to 
Mexican-style Spanish and validated in a pilot administration prior to the first year of 
data collection. Results from the validation exercise deemed both instruments to be 
reliable with a Cronbach's alpha of 0.88 for EGRA and 0.87 for EGMA (Barrios, 2014). 

EGRA has been used to assess reading skills of children at grades 1-3 in more 
than 50 countries and 70 languages (Gove and Wetterberg 2011). Prior studies showed it 
is a valid and reliable tool to measure reading achievement at early grade levels (RTI 
International 2009).’ EGRA includes eight subtests, including letter name knowledge, 


phonemic awareness, letter sound knowledge, listening comprehension, unfamiliar non- 


©The WWC sets the maximum acceptable rate of differential attrition at around 7-7.5 percent for studies 
with overall attrition lower than 10 percent (Deke, Sama-Miller and Hershey 2015). 

7 The tests have been criticized on the grounds that they measure a narrow set of skills (e.g. Bartlett et al., 
2015). Dubeck and Gove (2015) argue the tests measure foundational skills that are important in the early 
grades and have shown to correlate with later indicators of reading in standardized tests. 
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word reading, familiar word reading, passage reading and comprehension, and dictation. 
Trained examiners orally administer each test to children one-on-one. Completing the test 
takes 15-20 minutes. 

EGMA measures quantitative reasoning skills of students at grades 1-3. Prior 
studies showed that EGMA is a reliable measure of quantitative reasoning skills in 
primary schools (USAID 2010). EGMA has seven subscales, including number 
identification, quantity discrimination, missing number, word problem, 
addition/subtraction problem, shape recognition, and pattern extension. Trained 
examiners administer these tests to individual students orally. It takes 15-20 minutes to 
complete the test. 

Gains in EGRA/EGMA are measured by comparing results at the end of each 
corresponding grade per cohort. EGRA/EGMA are designed to capture the full range of 
abilities in 1st-3rd grade. Consequently, the average first grader is expected to score 
lower on the same test than the average third grader. 

Parent Perception Data 

A survey was administered to Cohort 2 parents to measure differences in opinions 
about their child's schooling and expectations. The survey was administered two years 
after baseline (in 2016). Of 91 parents who consented to respond to the parent survey at 
baseline, 83 took the survey two years later, a 91 percent response rate. 

Research Design and Empirical Strategy 

This evaluation follows an experimental design that takes advantage of lottery- 

based admissions into CHM. To estimate the causal effect of attending CHM on literacy 


and numeracy test scores, we estimate ANCOVA models that have the form: 
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(3) Vict =e + Gi +  Zic + Xie B + Eict 
where Yj¢¢ is an outcome for applicant 7 in applicant cohort c (2013, 2014, 2015), 
measured in post-treatment year ¢ (2014, 2015, 2016). This notation implies that we 
observe Cohort | over 3 waves of data collection, Cohort 2 over two waves, and Cohort 1 
over one wave; @, are cohort fixed effects; Z;, is an indicator that takes the value of one 
if applicant 7 wins the admissions lottery and zero otherwise; X;, is the vector of student 
background characteristics (including baseline cognitive scores and age); Because 
lotteries were stratified by gender, we include a gender dummy G;. To account for the 
fact that we have repeated post-treatment outcomes for the 2013 and 2014 applicant 
cohorts, we cluster the standard errors at the applicant level. Estimates of @ capture the 
regression-adjusted intent-to-treat (ITT) effect of being offered a first-grade slot at CHM 
at the lottery stage.® 

Because we observe different cohorts for different years, we follow Angrist et al. 
(2010) to convert each cohort's estimates into a normalized average per-year effect. We 
do this by modeling the impact of CHM on outcomes as a function of time spent in 
CHM: 

(4) Vict = Oe + Gi, + 0 Sict + Xic p + Eict 
where 6, are cohort fixed effects; s;., is the number of years applicant i from cohort c has 
spent in CHM at the time of the post-treatment measurement year ¢; X;, is the vector of 
student background characteristics (including baseline cognitive assessment scores); E;jct 


is aresidual. We estimate equation (2) using instrumental variables in a two-stage least 


8 Other research has shown initial offers are consistent and asymptotically normal estimators of the LATE, 
even if they have higher variance than other estimators (De Chaisemartin and Behaghel 2017). 
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squares (2SLS) framework in which the randomly assigned lottery outcome Z;, serves as 
an instrument for s;,;. Standard errors are clustered at the applicant level. 

A possible issue with randomization is that results may be sensitive to just a few 
outliers or clusters of observations. As described in Young (2018) and Athey and Imbens 
(2017), model-based regression designs may render cluster and robust variance estimates 
more unstable for a given regression design generating incorrect tests of significance. To 
counter this potential issue, we run randomization inference tests for each literacy and 
numeracy outcome as well omnibus tests of overall significance. 

EGRA/EGMA yield results in multiple domains (eight subscales for EGRA, and 
seven for EGMA). To ease presentation and interpretation we pool all subscale 
score/effects using a fixed-effects average effect size (Cooper, Hedges and Valentine 
2009; Bettinger, Kremer and Saavedra 2010; Katz, Kling and Liebman 2001). To 
produce a summary score for early grade literacy, and a summary score for early grade 
numeracy, we first standardize all domain score results relative to the control group mean 
and standard deviation (SD) in the first follow-up. Next, we calculate a precision- 
weighted average effect size. This is obtained using the inverse of the estimate's variance 
and calculating its t-statistic. This technique assumes estimating a single population 
parameter under a fixed-effects model (Cooper, Hedges and Valentine 2009). 
Reweighting for External Validity 

Families applying to the CHM lottery look very different from the average family 
in Mexico City. One concern is that our results cannot be easily extrapolated to the 
average family in Mexico City due to "participation bias" (Andrews and Oster 2018). To 


account for this, we use the method of "entropy weights" as proposed by Hainmuller 
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(2012) and Hainmueller and Xu (2013) to re-weight the sample to resemble the average 
Mexico City family. The method resembles propensity score matching. To the extent that 
most of the potential participation bias is explained by observable characteristics, results 
will be externally valid (Andrews and Oster 2018). 

To estimate the weights, we use Mexican household data from the Encuesta 
Nacional de Ingreso y Gasto de los Hogares 2014 (ENIGH-MCS). For this estimation we 
select households in Mexico City with children attending public schools in Ist-3rd grade. 
We use ENIGH variables that overlap with socioeconomic data collected from the CHM 
sample at baseline. The resulting weights closely replicate the first and second moments 
of the Mexico City household distribution. Results are shown in Table A.2. in the 
Appendix. 

Cost Data and Analysis 

To compare the costs of providing education at CHM compared to traditional 
public schools attended by control students, we use a version of the ingredients method, a 
simple yet rigorous procedure to guarantee inclusion of all relevant costs (Dhaliwal et al. 
2011; Levin and McEwan 2001, 2000). 

We collected cost data about CHM from school staff using the ingredients method 
excel worksheet in the fall of 2016. We collected costs only for the elementary school 
portion of CHM's operations. We pro-rated overlapping or other costs that serve the 
entire school by elementary school enrollment. 

To estimate costs in control schools we used information provided by the Mexico 


City Office of Education, the Mexican Ministry of Education and other published 
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sources. To fill in gaps and cross-reference the information provided by the Mexico City 
office of education we collected cost data from a sample of five control group schools. 

To estimate the average teacher cost in each of the control schools, we obtained 
data on the number of teachers and their seniority from the 2016-17 school census 
("Forma 911"). Using these data, we identified each of the teachers and other staff 
working in control group schools by type of position. We used published salary schedules 
to estimate compensation for each teacher and staff position, taking into account base 
salary and benefits, as well as any bonuses received by teachers participating in incentive 
programs such as "Carrera Magisterial." 

Most public schools in Mexico City received their buildings as donations from 
private individuals or foundations, and these donations go back many decades. The 
Mexican office of education was unable to provide information that would allow us to 
estimate infrastructure costs except in the case of one school, which was being rented at 
market rates. We pro-rated the rent paid by this one school on the basis of enrollment and 
applied that rate to other control schools based on enrollment. Utility costs (water, 
electricity) were obtained using data provided by AFSEDF for 83 public control group 
schools in our sample. 

To estimate the cost of providing breakfast and lunch in schools where this 
service is available, we use data contained in the school, teacher and student census of 
2013 ("CEMABE 2013") as well as budget information for the National Free Breakfast 
Program. We also use information from CEMABE 2013 to identify control group schools 
that participate in the national breakfast program. Costs were pro-rated by school 


enrollment. To estimate the cost of providing lunch in full-day schools, we use data 
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gathered from interviews with five control group schools to calculate per-student lunch 
costs. 

We gathered educational supply costs for textbooks and school supplies. 
Textbook costs were obtained using the published cost of producing these textbooks by 
the National Commission on Free Textbooks ("CONALITEG"). Mexico has a national 
free textbook program, and all students in public elementary schools receive textbooks 
free of charge. To estimate school supplies costs, we use published prices from one large 
supplier in Mexico City. The Mexican Ministry of Education publishes an approved 
"school supply" list that schools in Mexico City adhere to. This list was used as a 
benchmark to estimate costs for all control group schools based on their enrollment. 

Both CHM and traditional public schools receive donated goods and services. In 
addition, parents pay out-of-pocket costs. We ask questions in the parent survey about 
these expenses and report them below. One potentially important cost that we do not 
include is parental time, which is required at CHM and frequently donated at public 
schools across the country (Santibafiez, O' Donoghue and Abreu 2014). We do not have 
data to estimate the monetary value of parental time. Given that over 90 percent of the 
cost of public schools in Mexico covers teaching and non-teaching staff salaries we feel 
confident our estimation is close to the real cost of providing schooling to students in our 
sample. 

4. Results 
Effects on Early-Grade Literacy and Mathematics 
Table 4 shows intent-to-treat (ITT) effects of winning the admissions lottery. 


Results for the full sample (Table 4, Column 3) show that CHM students who won a seat 
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in the school's first grade cohort via lottery scored about 0.21 SD higher in early grade 
literacy, and 0.09 SD higher in early grade numeracy than students who did not win the 
lottery. These results represent an average for lottery Cohort 2013 students observed 
three times up to third grade; lottery Cohort 2014 students observed two times up to 


second grade, and lottery Cohort 2015 students observed once in first grade. 


Table 4. Effects of Private School on Learning Outcomes 


CQ) (2) (3) (4) (5) (6) 


Panel A. Early Grade Literacy (EGRA) 
Won admissions 
lottery 0.267" OLLOe* O.21%*> -030"* OT 0.18** 
(0.042) (0.036) (0.031) (0.042) (0.037) (0.031) 


Panel B. Early Grade Numeracy (EGMA) 
Won admissions 
lottery -0.03 O.13"* 0.09** 0.06* 0.10** 0.06 
(0.047) (0.044) (0.038) (0.038) (0.043) (0.038) 


Comparable Comparable 

Ist grade testing Ist grade testing 
Sample’! only conditions All Obs. only conditions All Obs. 
Demographic 
and cognitive 
baseline 
controls” No No No Yes Yes Yes 
N 239 328 502 239 328 502 


*** y<0.01, ** p<0.05, * p<0.1. Robust standard errors in parentheses. 

/1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, 
mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing 
values imputed, and missing dummies included for imputed cases. 

Note: All regressions include lottery strata fixed effects (gender and cohort). These are intent-to-treat effects with 
lottery offer as the key explanatory variable, and test outcome as the dependent variable. 


Column 1 of Results shown in Column 2 of Table 4 show that students who won 
the lottery for admission into CHM scored 0.19 SD higher than those who did not win the 


lottery, over the average time spent in CHM without taking into account 2015 — the year 
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where control students were tested in their homes. Unless otherwise noted, all of these 
effects are statistically significant at the 95 percent confidence level. The magnitude of 
the effects diminishes only slightly when student controls are included, to 0.17-0.30 SD 
in literacy, and 0.06-0.10 in mathematics. 

Figure 2 shows test score growth curves for each grade observed in the study. 
Results are pooled for all cohorts. They suggest that CHM improved learning outcomes 
in literacy in every grade except 2nd grade. In mathematics, CHM improved outcomes 
every grade, although the absolute levels are lower than they are for literacy. Results are 
statistically significant at the 95 percent confidence level, except for the Ist grade 


estimate for math. 
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Figure 2. Grade Level Growth Curves, All Cohorts 


Effect on Literacy (EGRA), by Grade, 2013 Cohort 


Difference between treatment and control Group, 95% conf. intervals 


4 
| 


Std. dev. over control group 


1 2 3 
grade 


Effect on Math (EGMA), by Grade, 2013 Cohort 


Difference between treatment and control Group, 95% conf. intervals 


Std. dev. over control group 


T T T 
1 2 3 
grade 


Note: These growth curves represent intent-to-treat with lottery offer as the key explanatory variable, and test outcome 
as the dependent variable. All regressions include lottery strata fixed effects (gender and cohort) and robust standard 
errors. Effect sizes are averaged using the meta-analytic technique discussed in the text. See text for more detail. 
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To account for differences across cohorts in the time spent at CHM, we model the 
causal effect of attending CHM on outcomes as a function of how much time students 
spend at the school. To do this we instrument time spent in CHM with lottery offers using 


a 2SLS approach. Table 5 shows the results. 


Table 5. Average Annualized Achievement Gains in Private School 


CQ) (2) (3) (4) (5) (6) 


Panel A. Effect of Winning the Lottery on Number of Years of Private School Attendance 


Won admissions lottery 1.70** 1.86** LS7** 
(0.104) (0.138) (0.131) 


Panel B. Average Annualized Achievement Gain in Literacy 
Years in CHM 
(instrumented) 0.15** Os10** O14: O17* <0.098"= °.0.10** 
(0.024) (0.019) (0.017) (0.023) (0.019) (0.016) 


Panel C. Average Annualized Achievement Gain in Math 


Years in CHM 
(instrumented) 0.02 0.07** 0.05** 0.05* 0.05** 0.03 
(0.028) (0.024) (0.021) (0.027) (0.023) (0.020) 
Comparable Comparab 
Ist grade testing Ist grade le testing 
Sample only conditions —_ All obs. only conditions All obs. 
Demographic and 
baseline controls’! No No No Yes Yes Yes 
N 239 328 502 239 328 502 


*** y<0.01, ** p<0.05, * p<0.1. Robust standard errors, clustered at the student level, are reported in parentheses. 

/1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, 
mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing 
values imputed, and missing dummies included for imputed cases. 

Note: These results are from regressions that instrument time spent in CHM with lottery offers. Models are estimated 
using 2SLS. All regressions include lottery strata fixed effects (gender and cohort). 


As expected, the first-stage results, shown in Panel A, indicate that winning an 


offer of admission results in about two additional years spent in CHM (full sample, Table 
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5, Column 3). The 2SLS results using the full sample indicate that students gain an 
average of 0.11 SD in literacy for each year spent in CHM and 0.05 in numeracy 
(Column 3, Table 5). When we restrict the sample to only the first follow-up (at the end 
of Ist grade), each additional year in CHM raises literacy scores by 0.15 SD and 
mathematics scores by 0.02 (not significant). When we take out 2015, the year when 
students were not tested under comparable conditions, each additional year in CHM 
raises literacy scores by 0.10 SD and mathematics scores by 0.07. Unless otherwise 
noted, all reported results are statistically significant at the 95 percent confidence level. 
Results change only slightly when demographic and cognitive baseline controls are 
included. 

To put this per-year effect in context, we calculate the average growth of the 
control group over the period under study. In EGRA, the control group grew an average 
of 0.45 SD per year in literacy and 0.71 SD per year in math. Relative to the control 
group, after two years the average student at CHM would be about one-half of a year 
ahead in literacy, and on par in math. 

The results presented above are robust to randomization inference in all samples 
except the Ist grade only sample in mathematics. Table 6 shows results from 
randomization inference omnibus tests performed on the joint significance of all pooled 
test score outcomes. The omnibus tests for randomization inference are significant at the 
95 percent confidence level for all samples of the literacy outcomes. The randomized p- 
value for the literacy average effect is 0.005, 0.001, and 0.001 for all three samples 
respectively. For the mathematics outcomes, the values are 0.14 for first grade only, and 


0.012 and 0.024 for the comparable testing conditions and all student samples. These 


29 


results imply that our findings are not sensitive to just a few outliers or clusters of 


observations as can sometimes be the case with smaller-sample studies. 


Table 6. Randomization Inference Tests of the Average Effect of Private School on 
Learning Outcomes 


Pooled 

Min Max Randomized 
Omnibus Tests p-value p-value p-value N 
Early Grade Literacy 
Sample of Ist grade students 0.000 0.001 0.005 239 
Sample of students in comparable testing conditions 0.000 0.001 0.001 328 
All students 0.000 0.001 0.001 502 
Early Grade Numeracy 
Sample of Ist grade students 0.136 0.138 0.137 239 
Sample of students in comparable testing conditions 0.012 0.013 0.012 328 
All students 0.022 0.024 0.024 502 


Note: This table presents results of randomization inference tests over the average effect size for the intent-to-treat 
estimates presented in Table 4. We use the randcmd in STATA to conduct these tests (Young, 2017). For each test, we 
conduct 1,000 permutations. 


Early-grade Gains Accounting for Differences in Length of School Day 

CHM's positive impact on early grade literacy and numeracy can be explained by 
a number of factors: length of the school day, better teachers, more parental involvement, 
health prevention services, better leadership, etc. We do not have data to unpack 
everything driving CHM's positive impact, but we do have data on one key factor: length 
of the school day. Our data allow us to identify whether Mexico City public schools 
attended by the control group are on a half-day (4-hour) or on a full-day (6 or 8 hour) 
schedule. With this information we can estimate the effects of CHM relative to other full- 
day schools and separate out the effect of the longer school day from the overall impact. 

One issue with this estimation is that while students enter CHM via lottery, 


parents do not follow a random process to enroll their students in public schools. Parents 
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could self-select into full-time public schools in ways that could bias an estimation of 


program effects. To explore this potential bias, we run descriptive statistics on students 


by whether they attend school full- or half-day. Results are shown in Table 7 and suggest 


balance on most characteristics, except for two. First, the Raven baseline score is 


statistically higher in the full-day school sample. This is not the case for the verbal WISC 


score. The magnitude of the Raven score difference, however, is small: about 0.02 SD. 


Second, single parents are more likely to seek out full time schools for their children. All 


other demographic characteristics show no significant differences among students 


attending full- and part-time public schools. 


Table 7. Characteristics of Control Group Students in Full-time vs. Half-day Public 


Schools 
Half-Day Full-Time 
Variable Public School Public School Diff. p-val 

Age (incoming 1st graders)’ 6.102 5.882 0.22 0.068 
(0.079) (0.080) (0.113) 

Raven Global Score (incoming Ist grade) 14.656 16.394 -1.739 0.001 
(0.343) (0.337) (0.484) 

WISC Global Score (incoming Ist grade) 24.63 25.123 -0.493 0.839 
(1.088) (0.865) (1.419) 

Monthly mean income” 3636.54 3370.08 266.46 0.208 
(187.47) (228.39) (292.85) 

Mother's education (Secondary=4) 4.18 4.14 0.040 0.480 
(0.067) (0.097) (0.095) 

Both parents are present in the home 0.59 0.37 0.221 0.001 
(0.041) -0.043 (0.06) 

Home is rented 0.412 0.348 0.064 0.333 
(0.041) (0.043) (0.060) 

Home has a kitchen 0.678 0.638 0.04 0.381 
(0.039) (0.043) (0.058) 

N I37 119 256 


1 Student Age for CHM is collected between March-June in the school year previous to entering Ist grade. Mexico 
City population, is collected between September-November of the corresponding school year. 


/2 Average monthly household income from wages and other fixed sources. 
Notes: Logistic regression. Indicator Full time schools = 1. Robust standard errors in parentheses. 
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To account for any bias that may arise from parental self-selection into full-time 
schools based on observable characteristics, we use propensity score matching. First, we 
restrict the sample to students who attend CHM and full-time public schools and fall 
within the common support region. To estimate the common support region and predict 
the propensity score we estimate a probit model of lottery status on baseline student 
covariates and lottery strata. We do this for the subset of schools that include CHM and 
full time public schools only. We then use the min/max method to eliminate observations 
that fall outside the common support region. Next, we estimate ITT using a matching 
estimator that accounts for the propensity to be selected for the program for students at 
CHM or full day public schools. We estimate the LATE using instrumental variables with 


the propensity score as an independent variable. Results are shown in Table 8. 


Table 8. Length of School Day as Possible Mediating Mechanism 


(1) (2) 
Panel A. Early Grade Literacy (EGRA) 
Won admissions lottery 0.10* 0.20** 
(0.055) (0.043) 
Years in CHM (instrumented) 0.03 0.05** 
(0.022) (0.018) 
Panel B. Early Grade Numeracy (EGMA) 
Won admissions lottery 0.11 0.07 
(0.068) (0.056) 
Years in CHM (instrumented) -0.02 -0.01 
(0.028) (0.023) 
Comparable testing 
Sample conditions All Observations 
Demographic and cognitive baseline controls! Yes Yes 
N 198 307 


*** y<0.01, ** p<0.05, * p<0.1. Robust standard errors in parentheses. 

Note: To obtain these results we first match CHM students to control-group students in full-time schools using 
propensity score matching. We keep students in the common support region. This eliminated 16 observations or 5% of 
the matched sample.. All regressions include lottery strata fixed effects (gender and cohort). 
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/1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, 
mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing 
values imputed, and missing dummies included for imputed cases. 


The literacy ITT on the matched full sample is similar as it was for the main 
model (0.20 SD vs. 0.21 SD). The normalized (instrumented) effect of attending CHM is 
much lower in the model that adjusts for length of school day, 0.05 SD vs. 0.11 SD for all 
students. In math, none of the effects are significant for the matched sample. Results are 
similar when we filter out the year when students were not tested under comparable 
conditions—except that now the literacy coefficient is only significant at the 10 percent 
confidence level, and the instrumented coefficient is no longer significant. Taken 
together, these findings indicate that the positive impact of CHM was largely mediated by 
the longer school day--the only mediating school-level variable that we could test. A 
portion of the effect does remain in literacy, suggesting that in this case other CHM 
characteristics remain important predictive factors.’ 

Re-Weighted Results - External Validity 

To explore whether our findings are sensitive to whether or not CHM lottery 
applicants are observationally similar to the average family in Mexico City, we re- 
estimate all models using entropy weights as described in the methods section. We use 
two sets of weights: (1) "average weights" based on the entire Mexico City household 
sample and (2) "low-income weights" using only households below the median income. 

Results are shown in Table 9. When the study sample is weighted to resemble the 
"average student" both the ITT and IV literacy effects of CHM are either negative 
? We conducted a small qualitative study of CHM and two control schools to understand what drove the 
differences between literacy and math. Results from classroom observations using the CLASS rubric 


indicate higher classroom quality scores in Spanish at CHM relative to control schools. In math, however, 
there was little observed difference (Razo 2017). 
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("comparable testing conditions" sample) or not statistically significant (full sample). The 
effects on math are not statistically significant across the board. When the study sample is 
weighted to resemble a low-income student (see Table 9, Columns 3 and 4), however, the 
results are once again positive and statistically significant for literacy. Results remain 
non-significant for math. 


Table 9. Effects of Private School Attendance on Achievement Re-weighted for 
External Validity 


(1) (2) (3) (4) 


Panel A. Early Grade Literacy (EGRA) 


Won admissions lottery 


(ITT) -0.18** 0.06 0.14** 0.28** 
(0.057) (0.074) (0.058) (0.048) 

Years in CHM 

(instrumented) (LATE) -0.10** 0.03 0.07** 0.12** 
(0.033) (0.042) (0.027) (0.023) 


Panel B. Early Grade Numeracy (EGMA) 


Won admissions lottery 0.00 -0.12 0.09 0.01 
(0.086) (0.076) (0.069) (0.058) 
Years in CHM 
(instrumented) (LATE) 0.01 -0.06 0.05 0.01 
(0.049) (0.044) (0.036) (0.030) 
Comparable Comparable 
testing testing 
Sample conditions Full Sample conditions Full Sample 
Calibrated to Calibrated to Calibrated to Calibrated to 
Type of Weight’! Average HH Average HH low-income HH__ low-income HH 
Demographic and cognitive 
baseline controls” Yes Yes Yes Yes 
Observations (N 328 502 328 502 


*** y<0.01, ** p<0.05, * p<0.1 
Note: All Samples includes only observations that are in the common support region attending CHM and other full- 
/1 Weights are calculated using ENIGH 2014 data. Variables include: income, rents, pub_elec, landphone, mother's 
education, and both parents present. Models are estimated using the ebalance command in STATA per Hainmuller & 
Xu (2013). See text for more details. 
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/2 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, 
mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing 
values imputed, and missing dummies included for imputed cases. 


This result is striking for two reasons. First, even though our results come from a 
single school, they can be replicated for a larger population of students who are 
observationally similar (1.e., students below the median income distribution in Mexico 
City). Second, CHM's model of education is most beneficial to low-income students-- 
their intended targeted population. When we give more weight to slightly better-off 
students so that the study sample resembles the average Mexico City household, CHM 
test scores are no longer above the control group test scores (and are even lower under 
some specifications). However, when we re-weight the study sample so that the lowest 
income students get a higher weight, the results are once again significant. Other authors 
have arrived at similar conclusions using charter school evidence from the United States 
(Chabrier, Cohodes and Oreopoulos 2016). 

Parent perceptions 

High parental involvement and expectations can positively influence academic 
outcomes for low-income students (Domina 2005; Marschall 2006; Yamamoto and 
Holloway 2010). Most research in the U.S. finds that charter school parents tend to be 
more satisfied and more involved in school than public school parents (Gleason et al. 
2010; Cheng and Peterson 2017). This is also the case at CHM (see Table 10). 

Parents of students who win the lottery of admission at CHM think more highly of 
their school than control parents, 0.97 points above the control group mean of 2.9 on a 
Likert-type scale where | is "Bad" and 4 is "Excellent" (Table 10, Column 2). CHM 


parents think their child's school is more demanding academically than parents whose 
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kids attend public schools (coefficient of 0.70). There does not appear to be a different in 
perception among parents with regard to how strict (discipline-wise) their child's school 
is--not surprising since public schools in Mexico are notoriously strict when it comes to 


enforcing order and discipline (Ochoa Cervantes and Diez-Martinez 2013). 


Table 10. Effects of Private School Attendance on Parent Perceptions 


Effects of Annualized 
Control Private Achievement 
Mean School (ITT) Gains (IV model) 
C) (2) (3) 

My school is very demanding academically 

(1-4) 2 0.70*** 0.44*** 
(0.188) (0.095) 

I think my school is (1=Bad, 4=Excellent) 2.9 0.97*** 0.62*** 
(0.152) (0.071) 

My school is very strict (discipline) (1-4) 3.8 0.71 0.45 
(1.192) (0.690) 

On a scale of 1-10, my school is a.... 8.2 125 [*** 0.96*** 
(0.275) (0.141) 

On a scale of 1-10, the probability my child 

will complete high school is .... 9.2 0.35* 0.22* 
(0.207) (0.116) 

On a scale of 1-10, the probability my child 

will complete college is .... 8.6 1.03*** 0.65*** 
(0.276) (0.150) 

Sample Cohort 2 Parents Only 

Demographic and baseline controls’! Yes Yes 

N 83 83 


*#EE D<0.01, ** p<0.05, * p<0.1. Note: All regressions include lottery strata fixed effects (gender and cohort). 
Robust standard errors are reported in parentheses. 

/1 Demographic and baseline test controls include: age, gender, cohort, RAVEN and WISC test scores at baseline, 
mother's education, single parent household, monthly household income, home is rented, home has a kitchen. Missing 
values imputed, and missing dummies included for imputed cases. 


CHM parents also tend to have higher expectations for their child's education than 
parents in the control group. On a scale of 1-10, control group parents set the probability 
of their child completing high school at 9.2. CHM parents set this probability 0.35 points 


higher, although the difference is statistically significant only at the 10 percent level. 
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Parents who win the lottery to attend CHM have significantly higher expectations than 
control group parents that their child will complete college, 1.03 points higher (on a scale 
of 1-10) over the control group mean of 8.6. 
7. Cost Analysis 

Table 11 shows operational costs of CHM relative to full- and half-day public 
schools in Mexico City. The monthly average cost of educating a student at CHM in US 
dollars is $240, or about US$2,400 per year. The estimated monthly cost of educating a 


student at a control group public school is US$120 or about US$1,200 per academic year. 


Table 11. Operating Costs of Private School and Control Public Schools (US $) /! 


Monthly Average Cost per Student 
Grades 1-6 (in $US 


Enrollment Payroll Other Total 

Type of School Teachers Total _ Costs Cost 

Treatment CHM 337 31.7 111.9 1284 240.3 
Control All Public Schools 361 43.3 69.0 53.7 P2251 
Half-Day att 36.5 55.9 26.7 82.6 

Full-Day (no lunch) 323 46.5 75.9 26.6 102.6 

Full-Day (lunch) 392 56.9 93.4 63.0 156.4 


Note: Exchange rate is 18 pesos per dollar. Costs are at current prices. 

/1The control group includes 83 public schools with available data. Source for CHM staff interviews and 
documentation provided by institution. Source for public schools is AFSEDF, the school census for 2016-17 
(Forma 911), and CEMABE 2013-14. See text for more details. 


Per-student costs are much higher at full-day than at half-day public schools. We 
estimate average per-student monthly costs to be US$156 (US$1,560 annually) at full- 
day schools offering lunch. Per-student average monthly cost at half-day schools is $82 
or about $800 per year. These estimated costs are in line with published sources. The 
national evaluation institute in Mexico, INEE, calculates an average annual (10 month) 
per-student cost of $900 for elementary public schools in the country (INEE, 2017). 


Teacher salaries do not drive the higher cost of CHM. In fact, the average teacher 
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payroll at CHM is about 27 percent lower than the teacher bill for control group public 
schools. At CHM, the vast majority of the per-student cost pays for infrastructure, 
equipment, materials and services offered at the school (including rent, utilities, 
extracurricular activities, books, school uniforms, lunch, breakfast, etc.). Thirty-three 
percent covers non-teaching staff (including administrators, psychologists, and support 
staff). Only 13 percent of the monthly per-student cost goes to pay teaching salaries. In 
the case of control public schools, 35 percent of the monthly average cost goes toward 
teacher salaries, 21 percent pays for non-teaching staff, and 44 percent covers 
infrastructure, utilities, equipment, materials and other services. 

Table A3 in the Appendix shows household expenses for CHM and control 
schools. Parents in CHM spend less out-of-pocket than parents in public schools in fees, 
school uniforms (which are required for public schools and CHM), textbooks, school 
supplies, and nutrition. The only category in which CHM parents spend significantly 
more money than public school parents is in transportation to and from school. 

8. Conclusion 

In low-income countries, private schools are perceived as superior alternatives to 
the public sector. The evidence suggests private schools can deliver high-quality 
education, at a fraction of the cost of public schools, in low-income countries (Patrinos, 
Barrera-Osorio and Guaqueta 2009; Muralidharan and Sundararaman 2015; Andrabi, Das 
and Khwaja 2008). We know less about the advantages private schools offer in middle- 
income countries to students who in some cases have a plethora of public school options. 
We use a lottery-based admissions process for one comprehensive private school model 


in Mexico City to test the claim that private schools benefit low-income children. 
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We find that children living in urban poverty in Mexico City perform better on 
academic tests of early grade literacy and mathematics when they attend a full-day, 
resource-rich private school as opposed to traditional public schools. Results are robust to 
various empirical strategies, including randomization inference. Parents of children in 
private school express higher levels of satisfaction with the school and are more likely to 
expect that their child will attend college than parents of children in public school. The 
private school model that CHM provides benefits low-income children the most--perhaps 
not surprising given that this is the intended target population of CHM. The costs of 
providing this education are significant: The school costs almost double what public 
schools in Mexico City cost, on a per pupil basis. This suggests private schools can be 
effective, but costly alternatives to improve the academic performance of vulnerable 
children in middle-income developing countries, such as Mexico. A test of whether these 
results hold if study participants are re-weighted to resemble the average household in 
Mexico City did not replicate these results. On the contrary, it suggests that private 
schools such as CHM may not provide any advantages to better-off children (or less poor 
children) above what traditional public schools provide. 

This study aims to fill an important gap in the literature examining private school 
impact on early-grade outcomes. One limitation of the study is that it focuses on early- 
grade basic literacy and mathematics skills. Although these foundational skills predict 
future academic outcomes (Duncan et al. 2007), they may not be able to capture the full 
range of benefits offered by a comprehensive school like CHM. Efforts to prepare 
children for college are likely to bear fruition in high school, one of the riskiest 


transitions for students in Latin America (Busso, Bassi and Mufioz 2013). Following this 
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sample of students over a longer period of time and collecting data on cognitive and non- 
cognitive outcomes, as well as on additional mediating factors that are policy relevant 
would greatly add to our understanding of whether and how private schools can benefit 


low-income populations in the region. 
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APPENDIX 


Table Al. Use of Time at CHM and Public Schools 


Traditional Jornada Tiempo 
CHM (half-day) Ampliada Completo 
Y) (full-day) _ (full-day) 
Daily schedule 7:00-17:00 8:00-12:30* 8:00-14:30 8:00-16:00 


Weekly Hours Spent on Various Activities 


Academic Activities (Total) 29 18 26 26 
Spanish 10 9 11 Il 
English as a second language 7 0 3 3 
Math 8 6 8 8 
Science 2 2 3 3 
Civics and Ethics 1 1 1 | 
Computers I 0 0 0 

Art education (Total) 2 1 2 2 

Physical Education (Total) 2 1 2 2 

Other activities (Total) 11 0 0 0 
Character/Leadership 1 0 0 0 
Hygiene/good habits 10 0 0 0 

Breakfast/Lunch (Total) 3.5 0 0 fie 

Recess (Total) 2.5 2.5 2 Z.5 

Total Hours per Week 50 22.5 32.0 40 


* Morning shift. Afternoon shift meets from 14:00 to 18:30 hrs. 
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Table A2. Descriptive Statistic Replication - Re-Weighted Sample 


All Weights (all HHS) 
total of weights: 221 
total of weights: 221 


Treated units: 221 
Control units: 241 


Before: without weighting 


rents 
bothpresent 

ing fijo 
mother_pri~s 
kitchen 

moth _educ_~m 
pub _electr~y 
landphone 


After: eweight_0 as the weighting variable 


rents 
bothpresent 


ing fijo 
mother_pri~s 
kitchen 

moth _educ_~m 
pub electr~y 
landphone 


Low-income 
Weights 

Treated units: 112 
Control units: 242 


Mean 
0.249 
0.765 
7870 
0.217 
0.941 
4.348 
1.000 
0.421 


Mean 
0.249 
0.765 


7870 
0.217 
0.941 
4.348 
1.000 
0.421 


Treatment - HH Data 


Variance 
0.188 
0.181 

38100000 
0.171 
0.056 
1.028 
0.000 
0.245 


Skewness 
1.162 
-1.248 
1.338 
1.372 
-3.750 
-0.082 


0.321 


Treatment - HH Data 


Variance 
0.188 
0.181 


38100000 
0.171 
0.056 
1.028 
0.000 
0.245 


total of weights: 112 
total of weights: 112 


Before: without weighting 


rents 
bothpresent 

ing fijo 
mother_pri~s 
kitchen 

moth _educ_~m 
pub electr~y 
landphone 


Mean 
0.277 
0.768 
3499 
0.259 
0.929 
4.116 
1.000 
0.304 


Skewness 
1.162 
-1.248 


1.338 
1.372 
-3.750 
-0.082 


0.321 


Treatment - HH Data 


Variance 
0.202 
0.180 

4774546 
0.194 
0.067 
0.914 
0.000 
0.213 


Skewness 
0.998 
-1.269 
-0.461 
1.101 
-3.328 
0.202 


0.854 


Mean 
0.421 
0.506 
3515 
0.156 
0.633 
4.186 
0.797 
0.350 


Mean 
0.249 
0.764 


7861 
0.218 
0.941 
4.347 
0.999 
0.420 


Mean 
0.421 
0.506 
3515 
0.156 
0.633 
4.186 
0.797 
0.350 


Control - CHM Data 


Variance 
0.240 
0.246 

6593485 
0.130 
0.229 
0.517 
0.159 
0.224 


Skewness 
0.324 
-0.025 
1.267 

1.912 
-0.558 
-0.023 
-1.494 
0.634 


Control - CHM Data 


Variance 
0.188 
0.181 

3810000 

0 
0.171 
0.056 
1.028 
0.001 
0.245 


Skewness 
1.163 
-1.244 


0.390 
1.367 
-3.734 
0.313 
-27.120 
0.323 


Control - CHM Data 


Variance 
0.240 
0.246 

6593485 
0.130 
0.229 
0.517 
0.159 
0.224 


Skewness 
0.324 
-0.025 

1.267 
1.912 
-0.558 
-0.023 
-1.494 
0.634 
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After: eweight_low as the weighting 


variable 
Treatment - HH Data Control - CHM Data 
Mean Variance Skewness Mean Variance Skewness 

rents 0.277 0.202 0.998 0.277 0.201 0.998 
bothpresent 0.768 0.180 -1.269 0.767 0.180 -1.262 
ing fijo 3499 4774546 -0.461 3496 4776145 0.387 
mother _pri~s 0.259 0.194 1.101 0.260 0.193 1.098 
kitchen 0.929 0.067 -3.328 0.928 0.067 -3.305 
moth educ ~m 4.116 0.914 0.202 4.114 0.916 0.764 
pub _electr~y 1.000 0.000 : 0.999 0.001 -34.440 
landphone 0.304 0.213 0.854 0.304 0.212 0.854 


Note: These are estimated using the entropy weights method and the ebalance command in STATA per 
Hainmuller & Xu (2013). See text for more details. 


Table A3. Parent Out-of-Pocket Expenses, (US $) 


Public (Control) 

CHM Schools 
Lump-sum costs (one time) 
At the beginning of the school year, how much did you pay in 
matriculation fees? $0 $4 
For this school year, how much did you pay for school 
uniforms? $0 $37 
At the beginning of the school year, how much did you pay for 
textbooks and school materials? $0 $21 
For this school year, how much did you pay for other school 
expenses? $10 $16 
Recurrent costs 
Lunch and/or breakfast at school? $1 $142 
Transportation to and from school? $217 $153 
School materials, supplies, copies, etc.? $20 $60 
School tuition, and/or fees? $75 $84 
Total Per-Student Average Out-of-Pocket Expenses (per year) $324 $519 


Notes: These are responses for the parents of Cohort 2014 students. Survey was administered in June of 
2016 (end of the school year). Parents were asked to refer to the study sample child only, when responding 
questions. Parent responses were coded in the column (CHM or Public School) corresponding to their 
actual child's enrollment (not lottery-status). This was done in order to get accurate estimates of average 
costs in CHM vs. public schools. As explained in the text, the vast majority (94%) of students in our 
sample complied with their assigned lottery status, so the numbers here accurately reflect what parents in 
treatment/control groups were paying out of pocket. All expenses were converted to US$ using an 
exchange rate of 20 Mexican Pesos per US$. Monthly expenses such as transportation, lunch, school 
materials, tuition and fees, were converted to yearly costs using a 10-month academic school year. Other 
school expenses include: school events and festivities, field trips, school carnival, school pictures, extended 
day fees, and music lesson fees. 


47 


